Real-World Evidence: Understanding Sources of Variability Through Empirical Analysis

In this issue of Value in Health, Thompson1Thompson D. Replication randomized, controlled trials using real world data: what could go wrong?.Value Health. 2021; 24 (In Press)Abstract Full Text PDF PubMed Scopus (5) Google Scholar reviews the merits and potential pitfalls efforts to replicate randomized (RCTs) real-world data (RWD). We agree with much content article but disagree author’s primary conclusion that there is only downside emulation activities because failure emulate will undermine confidence real-world-data (RWD) studies. Indeed, objectives studies are identify sources variability RWD, understand its limitations advantages, thereby appreciate appropriate utility value. There many reasons why RCTs might generate different treatment effect estimates than observational fact, it somewhat surprising would ever be able account for sufficient confounders results a RCT. But an empirical question, central number RCT emulations currently underway. Randomization controls bias from both observed unobserved confounders. With exception instrumental variables techniques, most statistical methods used control Thus, efficacy–effectiveness gap largely amounts whether important known can measured available databases; likely vary by source, disease state, eligibility criteria, outcome measures, others. Bartlett et al2Bartlett V.L. Dhruva S.S. Shah N.D. Ryan P. Ross J.S. Feasibility clinical trial evidence.JAMA Network Open. 2019; 2 (e1912869)Crossref (80) found cross-sectional review 220 published high-impact journals 2017, 15% potentially emulated administrative claims or electronic health records (EHR). Despite have similar average RWD,3Anglemyer A. Horvath H.T. Bero L. Healthcare outcomes assessed study designs compared those trials.Cochrane Database Syst Rev. 2014; MR000034PubMed Scholar, 4Concato J. N. Horwitz R.I. Randomized, trials, studies, hierarchy research designs.N Engl J Med. 2000; 342: 1887-1892Crossref (2548) 5Benson K. Hartz A.J. A comparison trials.N 1878-1886Crossref (1811) other documented wide variation RWD within same therapeutic areas,6Madigan P.B. Schuemie M. al.Evaluating impact database heterogeneity on results.Amer Epidemiol. 2013; 178: 645-651Crossref (110) including propensity score-based methods.7Forbes S.P. Dahabreh I.J. Benchmarking analyses against trials: assessing score methods.J Gen Intern 2020; : 1-9Google Before RWE reliably regulatory decision making as required 21st Century Cures Act,8US Food Drug AdministrationUse evidence support decision-making medical devices: guidance industry Administration.https://www.fda.gov/regulatory-information/search-fda-guidance-documents/use-real-world-evidence-support-regulatory-decision-making-medical-devicesDate accessed: November 8, 2020Google need better understood. Those include not included themselves (eg, nonrepresentativeness dataset) also approaches—and assumptions—of scientists who choose which interrogate questions. One promising approach understanding these compare obtained attempt endpoints, features closely possible. small attempted findings RCTs,9Seeger J.D. Bykov Bartels D.B. al.Safety effectiveness dabigatran warfarin routine care patients atrial fibrillation.Thromb Haemost. 2015; 114: 1277-1289Crossref (105) and, similarly, several been confirmed RCTs.10Schneeweiss S. Seeger Landon Walker A.M. Aprotinin during coronary-artery bypass grafting risk death.N 2008; 358: 771-783Crossref (268) Scholar,11Noseworthy P.A. Gersh B.J. Kent D.M. al.Atrial fibrillation ablation practice: CABANA generalizability.Eur Heart 40: 1257-1264Crossref (58) some instances, disagreements between were attributable avoidable errors design.12Hernán M.A. Alonso Logan R. al.Observational analyzed like experiments: application postmenopausal hormone therapy coronary heart disease.Epidemiology. 19: 766Crossref (471) This has led focus importance design attempting draw causal inferences regarding effects.13Petersen M.L. van der Laan M.J. Causal models learning integrating modeling estimation.Epidemiology. 25: 418-426Crossref (62) 14Goodman S.N. Schneeweiss Baiocchi Using thinking differentiate useful misleading research.JAMA. 2017; 317: 705-707Crossref (43) 15Franklin J.M. When how substitute trials?.Clini Pharmacol Ther. 102: 924-933Crossref (102) However, conditions under possible obtain may differ when disagreement. The such Randomized Controlled Trials Duplicated Prospective Longitudinal Insurance Claims: Applying Techniques Epidemiology (RCT DUPLICATE) Observational Patient Evidence Regulatory Approval uNderstanding Disease (OPERAND) influence ability close gap. Are certain states more complete others? How influenced type versus records, combination record data; neither enhanced linkage pharmacy data)? estimate effects data? researchers about they inclusion exclusion criteria trial, measure select methods, comparing groups? questions mind, emphasize various primarily concerned demonstrating agreement studies—or usefulness (RWE). Rather, seek cause differs designed answer question patient population. As Thompson notes, DUPLICATE16RCT DuplicateRandomized duplicated prospective longitudinal insurance claims: applying techniques epidemiology.https://www.rctduplicate.orgDate large-scale, FDA-funded approximately 30 data. For where mimic data, work provide information across categories. OPERAND, hand, providing complementary insights variation. differs: OPERAND involves independent analysis teams initial large dataset. Differences analytic decisions considered, prespecification methodological approaches. Further, evaluating broader population linked de-identified EHR enables expansion endpoints HbA1c, pain scores, lipid levels) typically captured Finally, loosening inclusion/exclusion specified replicated affects estimates. actually treated practice narrower group studied trials? It through rigorous value revealed. Although DUPLICATE highly structured their attempts criticized being planned conducted. gives conducting target “fit analysis” so result. criticism impugn RWD; rather, posteriori fit falsely substantiate RWD. approaches minimize criticism: preregistration intended helpful.17Zarin D.A. Crown W.H. Bierer B.E. Issues registration studies.J Clin 121: 29-31Abstract (2) More convincing provided if completed before known, accomplished conducted products already market. An example prediction was trial,18Packer D.L. Mark Robb R.A. al.Effect catheter vs antiarrhythmic drug mortality, stroke, bleeding, cardiac arrest among fibrillation: trial.JAMA. 321: 1261-1274Crossref (445) simultaneously mimicking trial.11Noseworthy very similar. effort includes predicted prior release results. Other work, funded US Administration (FDA), underway.19Yale School MedicineCenter Outcomes Research Evaluation (CORE).https://medicine.yale.edu/core/current_projects/cersi/research/Date concurrent validation RWE, revealing well choices, measures correlate RCT, then usefully relied upon subsequent subpopulations). true FDA strong preference decisions, agency legislative mandate use making. derived safety approved FDA’s Sentinel Initiative.20SentinelAbout (FDA) Initiative.https://www.sentinelinitiative.org/aboutDate Act extend new indications previously drugs. framework nicely outlines program.21US AdministrationFramework Real-World Program.https://www.fda.gov/media/120060/downloadDate clear considers based settings, does address extant routinely collected purposes. carefully yield variability, limitations, RWE. trust subjecting scientific rigor defined reinforce validate, undermine, trustworthiness Author Contributions: Concept design: Crown, Drafting manuscript: Critical revision paper intellectual content: Obtaining funding: Administrative, technical, logistic support: Conflict Interest Disclosures: Drs reported receiving grants Amgen, AstraZeneca, Merck, Optum, Pfizer, Sanofi, UCB BioSciences outside submitted work. Funding/Support: authors received no financial research. Data: What Could Go Wrong?Value HealthVol. 24Issue 1PreviewWith growing interest (RWE) purposes, policy makers considering best assess credibility Because (RCT) long regarded gold standard high-quality research, one pursued see extent nonrandomized If congruent, reasoning goes, bolster underlying validity generated. Full-Text